Adult Website Classifier

The goal of this project was to detect adult websites and pages that are not safe for kids. We use five different techniques. We create an adult vocabulary and use a composite classifier formed of multiple Naïve Bayes classifiers to classify pages based on url, title, keywords and content. We use hue, saturation and histogram of gradients to train random forests, different boosting classifiers and MLP on boxed images for local image classification. The intent is to then use a Viola-Jones or Haar approach to classify images globally. We show an edge-detection technique that works better than Canny’s for some images. We show an extension to Markov chains that can help detect edges. The intent is to use classifiers such as SVMs with Gaussian kernels to use edge information in detecting body parts. Lastly, we propose AdultRank, a ranking metric that serves as an indicator of the adultness of a page. All the techniques together can be used effectively to detect adult web sites and pages. The only overlap this work has with previous related work is in image recognition using the features we have used and edge detection techniques. Introduction Website classification is an old problem. Internet Explorer labels websites as phishing and malware. Google leaves out malware sites from its search results. The goal of this project was to build a classifier that can classify websites and web pages as adult, i.e. sites and web pages that are unsafe for kids. Applications of this classification are many. Parents don’t want their kids exposed to adult content. Some adults find porn images offensive. Some governments ban porn sites and have an ongoing requirement to detect them. Many porn sites have a malware payload and install rootkits, adware, spyware and other viruses, so guarding against them is an additional safety measure. In general, sites and pages could be classified as adult sites based on many factors such as adult images, sexual content, violent content, racist sentiments, extreme radical views etc. The scope of this project is limited to the first two categories. We try to classify pages based on metadata and try to find good classifiers for adult images. Video classification was out of scope for this project but can be done by analyzing individual frames. We also extend Markov’s model, propose a ranking metric AdultRank and propose a new convolution filter for edge detection. Strategy We employ three main techniques: image analysis, text analysis and ranking. For text analysis, we inspect page title, keywords, url and content. For image analysis, we use different image recognition techniques. The OpenCV package was used for image recognition and ML classifiers. For ranking, we propose AdultRank, a ranking metric similar to PageRank. 1 Url Text Classification 1.1 Url Features The following metadata of web pages were used for adult classification:  Meta tag: “rating”. There are some standards that sites can use to indicate adult content but none that we saw use the “rating” meta tag. If the rating tag is found to be adult or restricted, the page is classified as adult. The code was not included in the final toolset since this analysis can be done independently without using machine learning techniques.  Title: if the page title contains an adult word, the page is classified as adult.  Url: if the page title contains an adult word, the page is classified as adult. The chosen implementation is naïve: it looks if the words from an adult dictionary exist in the url as substrings. A proper implementation would parse the url into words with an optimum match, take the site content into consideration and then do a dictionary lookup. As an example, we consider tit to be an adult word but this false positives sites with “title” in the url. As an example of requiring site content to be considered, “google” can be parsed as the mathematical number google, and “go” “ogle” – the site content would help determine which of the two parsings is more appropriate. 